Imaging display device and wearable device

ABSTRACT

An imaging display device includes an imaging unit, a processing unit, a display unit, and a pupil detection unit. The imaging unit includes a plurality of photoelectric conversion elements and is configured to acquire first image information. The processing unit is configured to process a signal from the imaging unit and generate second image information. The display unit is configured to display an image that is based on the signal from the processing unit. The pupil detection unit is configured to detect vector information of a pupil. The processing unit generates the second image information by processing the first image information based on the vector information on the pupil.

BACKGROUND Field

One disclosed aspect of the embodiments relates to an imaging displaydevice and a wearable device.

Description of the Related Art

Wearable devices, which are called, for example, a head mounted displayand smart glasses, including an imaging display device have been known.In a system used in such a wearable device, a scenery in front of a useris captured as an image using an imaging apparatus, and the image isdisplayed on a display device. By the system, the user can feel as ifthe user is directly seeing the scenery in an external world while theuser sees the scenery through the display device. If there is a largedifference between an image displayed on the display device and an imageof the external world, the user feels uncomfortable or feels sick. Thus,research and development for reducing the difference have been widelyconducted.

Japanese Patent Application Laid-Open No. 2004-222254 discusses atechnique of generating image information on an image captured at thecenter position of a lens of glasses, from image information obtained byimage capturing using a plurality of image sensors arranged at a glassesframe.

While the technique discussed in Japanese Patent Application Laid-OpenNo. 2004-222254 generates image information on an image captured at thecenter position of the lens of glasses, a positional relationshipbetween the lens of glasses and a pupil of a user is not factored in. Ina case where there is a large difference between a central axis of apupil and the center position of a lens of glasses, a difference betweena captured image and a real image can occur. Particularly in an imagingdisplay device including a display unit, a difference is generatedbetween a display image and a real event, and therefore the user mightfeel uncomfortable.

SUMMARY

According to an aspect of the embodiments, an imaging display deviceincludes an imaging unit, a processing unit, a display unit, and a pupildetection unit. The imaging unit includes a plurality of photoelectricconversion elements, and is configured to acquire first imageinformation. The processing unit is configured to process the firstimage information from the imaging unit and generate second imageinformation. The display unit is configured to display an image that isbased on the second image information from the processing unit. Thepupil detection unit is configured to acquire vector information on apupil. The processing unit generates the second image information byprocessing the first image information based on the vector informationon the pupil.

Further features of the disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A, 1B, and 1C are schematic diagrams illustrating an imagingdisplay device according to a first exemplary embodiment.

FIGS. 2A, 2B, 2C, and 2D are schematic diagrams illustrating the imagingdisplay device according to the first exemplary embodiment.

FIG. 3 is a table illustrating an operation of the imaging displaydevice according to the first exemplary embodiment.

FIGS. 4A, 4B, 4C, 4D, and 4E are schematic diagrams illustrating animaging display device according to a second exemplary embodiment.

FIGS. 5A, 5B, and 5C are schematic diagrams illustrating an imagingdisplay device according to a third exemplary embodiment.

FIGS. 6A and 6B are schematic diagrams illustrating an imaging displaydevice according to a fourth exemplary embodiment.

FIGS. 7A and 7B are schematic diagrams illustrating an imaging displaydevice according to a fifth exemplary embodiment.

FIGS. 8A, 8B, and 8C are schematic diagrams illustrating the imagingdisplay device according to the fifth exemplary embodiment.

FIGS. 9A and 9B are schematic diagrams illustrating an imaging displaydevice according to a seventh exemplary embodiment.

FIG. 10 is a schematic diagram illustrating a wearable device.

FIGS. 11A, 11B, and 11C are schematic diagrams illustrating an imagingdisplay device according to a ninth exemplary embodiment.

FIG. 12 is a schematic diagram illustrating the imaging display deviceaccording to the ninth exemplary embodiment.

FIGS. 13A, 13B, and 13C are schematic diagrams illustrating an imagingdisplay device according to a tenth exemplary embodiment 0.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, exemplary embodiments will be described with reference tothe drawings. In the description of each exemplary embodiment, thedescription of the same configurations as those in another exemplaryembodiment will be omitted in some cases. In addition, the exemplaryembodiments can be appropriately changed or combined.

A first exemplary embodiment will be described with reference to FIGS.1A, 1B, 1C, 2A, 2B, 2C, and 2D. FIG. 1A is a schematic diagramillustrating an imaging display device 100 according to the presentexemplary embodiment. The imaging display device 100 includes an imagingunit 101, a processing unit 102, a display unit 103, and a pupildetection unit 104.

The imaging unit 101 includes a plurality of light receiving elements.For example, the light receiving element is a photoelectric conversionelement, and performs an image capturing operation of converting lightemitted from the outside (external information), into an electronicsignal and acquiring image information. The pupil detection unit 104includes a plurality of light receiving elements. For example, the lightreceiving element is a photoelectric conversion element, converts lightinto an electronic signal, and detects pupil information. The pupilinformation at least includes vector information between the pupildetection unit 104 and a pupil, and may include the size of the pupiland information regarding a line of sight. The vector informationincludes a distance between the pupil and the pupil detection unit 104,and a direction from the pupil detection unit 104 to the pupil, forexample. The processing unit 102 generates image information(hereinafter, referred to as pupil-based adjusted image information)obtained by adjusting image information from the imaging unit 101 usingpupil information obtained from the pupil detection unit 104. Thedisplay unit 103 includes a plurality of light emitting elements. Theplurality of light emitting elements converts light into an electronicsignal. The display unit 103 displays (outputs) an image correspondingto the pupil-based adjusted image information generated by theprocessing unit 102. In the imaging unit 101 and the display unit 103,it can also be said that a plurality of pixels is arranged in an array.Each pixel of the imaging unit 101 includes at least one light receivingelement, and each pixel of the display unit 103 includes at least onelight emitting element. The processing unit 102 receives imageinformation from the imaging unit 101 and outputs pupil-based adjustedimage information to the display unit 103. The processing unit 102 canalso output a control signal of an image capturing operation to theimaging unit 101, and output a control signal of a display operation tothe display unit 103.

FIG. 1B is a schematic diagram illustrating a modified example of theimaging display device 100 according to the present exemplary embodimentthat is illustrated in FIG. 1A. The processing unit 102 of an imagingdisplay device 120 can communicate with a processing device 105. Theprocessing unit 102 and the processing device 105 connect with eachother via a network. The processing device 105 is provided on theoutside of the imaging display device 120, and may be provided on acloud, for example. The processing unit 102 and the processing device105 exchange information with each other, and generate pupil-basedadjusted image information from image information and pupil information.In FIG. 1B, image information obtained by image capturing using theimaging unit 101 is converted into pupil-based adjusted imageinformation by the processing unit 102 that has obtained informationfrom the processing device 105. In this manner, the imaging displaydevice 120 can generate pupil-based adjusted image information usinginformation accumulated in an external device.

FIG. 1C is a schematic diagram illustrating a modified example of theimaging display device 100 according to the present exemplary embodimentthat is illustrated in FIG. 1A. The processing unit 102 of an imagingdisplay device 130 communicates with a processing device 106, and theprocessing device 106 further communicates with another processingdevice 105. The processing device 106 is on a cloud and performs dataaccumulation, for example. The processing unit 102 and the processingdevice 105 connect with each other via a network, and the processingdevice 106 and the processing device 105 connect with each other via anetwork. In FIG. 1C, the processing unit 102 receives settinginformation accumulated in the processing device 106, and generatespupil-based adjusted image information based on the setting information.The setting information includes various values for generatingpupil-based adjusted image information, such as body information on theuser and basic information regarding an environment and a target object.The processing unit 102 also transmits a plurality of pieces ofinformation including image information from the imaging unit 101, tothe processing device 106. The plurality of pieces of information istransmitted to the processing device 105 via the processing device 106.Based on the plurality of pieces of received information, the processingdevice 105 generates various values for generating pupil-based adjustedimage information, and transmits the generated various values to theprocessing device 106. The processing device 106 updates basicinformation and various values that are accumulated therein, and holdsthe updated basic information and values as new information. In thismanner, the imaging display device 130 can generate pupil-based adjustedimage information using information accumulated in an external device.

The pupil detection unit 104 of the imaging display device according tothe present exemplary embodiment will now be described with reference toFIGS. 2A, 2B, 2C, and 2D. FIGS. 2A, 2B, 2C, and 2D are schematicdiagrams illustrating the imaging display device according to thepresent exemplary embodiment. FIGS. 2A, 2B, 2C, and 2D illustrate a casewhere the imaging display device 100 illustrated in FIG. 1A includes aneyewear-shaped casing. While FIGS. 2A, 2B, 2C, and 2D illustrate a casewhere the imaging display device 100 includes two display units 103, theimaging display device 100 may have a configuration including onedisplay unit 103 and one imaging unit 101. As illustrated in FIGS. 2A,2B, 2C, and 2D, the pupil detection unit 104 at least includes a pupildetection unit 1041. The pupil detection unit 1041 acquires vectorinformation between a pupil and the pupil detection unit 1041. Based onthe vector information obtained by the pupil detection unit 1041, theprocessing unit 102 generates vector information between the pupil andthe imaging unit 101. Then, using the vector information between thepupil and the imaging unit 101, the processing unit 102 performsadjustment to eliminate a difference in spatial position of a targetobject that can be generated between image information and a real image.The processing unit 102 adjusts image information acquired by theimaging unit 101 and generates pupil-based adjusted image information.By using the pupil-based adjusted image information, a display imagejust like a real image can be displayed on the display unit 103. All ofFIGS. 2A, 2B, 2C, and 2D illustrate a case where a central axis C1 of acertain pupil coincides with the center of one display unit 103 facingthe corresponding pupil. A range R indicates a range in which imagecapturing is performed by the imaging unit 101.

In FIG. 2A, the imaging unit 101 and the pupil detection unit 1041 arearranged on the central axis C1 of the pupil. The pupil detection unit1041 acquires vector information P1 between the pupil and the pupildetection unit 104. The image information acquired by the imaging unit101 is adjusted based on the vector information P1. Since the pupildetection unit 1041 and the imaging unit 101 are arranged on the centralaxis C1, adjustment is performed factoring in just a distance betweenthe pupil and the imaging unit 101 that is included in the vectorinformation P1. Accordingly, a processing load on image information isreduced and a processing time can be shortened. Power consumption can bealso reduced.

In FIG. 2B, both of the imaging unit 101 and the pupil detection unit1041 are not on the central axis C1 of the pupil. More specifically, theimaging unit 101 and the pupil detection unit 1041 are arranged outwardfrom the central axis C1 by a predetermined distance in an X direction.The pupil detection unit 1041 is on a central axis C2 of the imagingunit 101. The pupil detection unit 1041 acquires the vector informationP1 between the pupil and the pupil detection unit 104. The processingunit 102 preliminarily holds vector information between the pupildetection unit 1041 and the imaging unit 101. Image information acquiredby the imaging unit 101 is adjusted based on the vector information P1.The vector information P1 includes a distance from the central axis C1to the central axis C2, and a distance from the pupil to the pupildetection unit 104. Since the imaging unit 101 and the pupil detectionunit 1041 are positioned on the same central axis C2, vector informationbetween the pupil and the imaging unit 101 can be easily generated frominformation obtained by the pupil detection unit 1041. As compared withthe case illustrated in FIG. 2A, a freedom degree in the arrangement ofanother portion such as the display unit 103 increases, and this cancontribute to the miniaturization of the imaging display device.

In FIG. 2C, while the pupil detection unit 1041 is arranged on thecentral axis C1 of the pupil, the imaging unit 101 is not arranged onthe central axis C1 of the pupil. The central axis C2 of the imagingunit 101 is arranged outward from the central axis C1 by a predetermineddistance in the X direction. The processing unit 102 preliminarily holdsvector information P2 between the pupil detection unit 1041 and theimaging unit 101. The pupil detection unit 1041 acquires the vectorinformation P1 between the pupil and the pupil detection unit 1041.Based on the vector information P2 and the vector information P1, theprocessing unit 102 adjusts image information acquired by the imagingunit 101, and acquires pupil-based adjusted image information.

In FIG. 2D, while the imaging unit 101 is arranged on the central axisC1 of the pupil, the pupil detection unit 1041 is not arranged on thecentral axis C1 of the pupil. A central axis C3 of the pupil detectionunit 1041 is arranged outward from the central axis C1 by apredetermined distance. In this configuration, the processing unit 102preliminarily holds the vector information P2 including a distancebetween the imaging unit 101 and the pupil detection unit 1041. Thepupil detection unit 1041 acquires the vector information P1 including adistance from the pupil. Based on the vector information P2 and thevector information P1, the processing unit 102 adjusts image informationacquired by the imaging unit 101, and acquires pupil-based adjustedimage information.

As illustrated in FIGS. 2A, 2B, 2C, and 2D, by using the pupil detectionunit 1041, image information obtained by the imaging unit 101 can bedisplayed on the display unit 103 just like an actually-viewed image. Aplurality of pupil detection units 1041 may be provided, and one pieceof pupil information with high accuracy may be generated from aplurality of pieces of pupil information. The vector information inFIGS. 2A, 2B, 2C, and 2D has been described to include information inthe X direction and a Y direction, but actually, the vector informationcan include information in a sheet surface depth direction, that is tosay, information in a Z direction.

Next, while comparing with another example, the imaging display deviceaccording to the present exemplary embodiment will be described withreference to FIG. 3. FIG. 3 is a table illustrating an operation of theimaging display device according to the present exemplary embodiment.For the sake of simplifying the description, the description will begiven with reference to FIG. 3 using a part of the configurationsillustrated in FIGS. 2A, 2B, 2C, and 2D. Specifically, only a portion ofglasses that corresponds to one eye is extracted from the eyewear-shapedimaging display device.

FIG. 3 illustrates configurations of Examples 1 to 3, a real image or adisplay image displayed on the display unit 103, and image informationacquired by the imaging unit 101. Example 1 illustrates a case where atarget object is actually viewed by an eye, and Examples 2 and 3illustrate a case according to the present exemplary embodiment.

First of all, Example 1 illustrates a case where a target object isactually viewed by an eye. In a positional relationship between a pupiland a target object in this example, the target object is on the centralaxis C1 of the pupil. As illustrated in a real image in Example 1, it isrecognized that the target object is on the central axis of the pupil,the target object is at a position distant by a predetermined lineardistance, and a size of the target object is a predetermined size.

Example 2 illustrates a case according to the present exemplaryembodiment. In Example 2, while the target object is on the central axisC1 of the pupil, the imaging unit 101 is arranged outward from thecentral axis C1 by a predetermined distance in the X direction. In otherwords, in Example 2, the imaging unit 101 is not arranged on the centralaxis C1 of the pupil. Accordingly, in image information acquired by theimaging unit 101, the target object is shifted in a plus direction ofthe X direction. Furthermore, because image capturing is performed at aposition closer to the target object by a distance between the pupil andthe imaging unit 101, in the image information acquired by the imagingunit 101, a subject becomes larger than that in the real image. InExample 2, since adjustment is performed in a manner such that imagecapturing of the target object is performed on the central axis C1 ofthe pupil, the target object is displayed at the center in a displayimage, and an image close to the real image is obtained.

Example 3 illustrates a case of the imaging display device according tothe present exemplary embodiment. In Example 3, the captured image inExample 2 is further adjusted based on a distance between the imagingunit 101 and the pupil. In Example 3, similarly to Example 2, while thetarget object is on the central axis C1 of the pupil, the imaging unit101 is arranged outward from the central axis C1 by a predetermineddistance. In other words, in Example 3, the imaging unit 101 is notarranged on the central axis C1 of the pupil. Accordingly, in imageinformation acquired by the imaging unit 101, the target object isshifted rightward with respect to the sheet surface. Furthermore,because image capturing is performed at a position closer to the targetobject by a distance between the pupil and the imaging unit 101, in theimage information acquired by the imaging unit 101, a subject becomeslarger than that in the real image. In Example 3, because the imageinformation obtained by the imaging unit 101 is adjusted based on thepositions of the pupil and the imaging unit 101 and the distance betweenthe pupil and the imaging unit 101. More specifically, as illustrated ina display image in Example 3, the target object is displayed in such amanner that the target object is on the central axis C1 of the pupil,the target object is at a position distant by a predetermined lineardistance, and a size of the target object is a predetermined size.Accordingly, as compared with the image information acquired by theimaging unit 101, the target object is at the center, the target objectis positioned at a distance, and the target object is displayed in asmaller size, in the display image. It can be seen that, as comparedwith the image information acquired by the imaging unit 101 in Example2, the display image in Example 3 is similar to the real imageillustrated in Example 1.

As described above, the user can desirably use the imaging displaydevice according to the present exemplary embodiment without feelinguncomfortable. In an imaging unit and an eyewear-shaped imaging displaydevice, for example, since the position of a pupil varies depending onthe user, if a display image is generated based on vector informationbetween the imaging unit and the center of the imaging display devicecorresponding to one eye, a difference can be generated between a realimage and the display image. Using the imaging display device accordingto the present exemplary embodiment, a highly-accurate display image canbe generated using a pupil detection unit, and a display image similarto a real image can be generated.

Next, a structure of the imaging display device 100 will be described.First of all, the photoelectric conversion element included in theimaging unit 101 can include a photodiode and a photoelectric conversionfilm, for example. Examples of material of the photodiode includesilicon, germanium, indium, gallium, and arsenicum. Examples of the typeof the photodiode include a PN junction photodiode, a PIN photodiode,and an avalanche photodiode.

For example, a complementary metal-oxide semiconductor (CMOS) imagesensor can be used as the imaging unit 101, and the CMOS image sensormay be a front-side illumination CMOS image sensor or a backsideillumination CMOS image sensor. In addition, the CMOS image sensor mayhave a stack structure of a semiconductor substrate on which aphotodiode is arranged, and a semiconductor substrate on which ascanning circuit and a control circuit are arranged.

As the material of the photoelectric conversion film, there are organicmaterial and inorganic material. For example, an organic photoelectricconversion film has a structure including at least one organic layer forphotoelectric conversion between a pair of electrodes. An organicphotoelectric conversion film may have a structure in which a pluralityof organic layers is stacked between a pair of electrodes. An organiclayer may be made of single material or made of a plurality of mixedmaterials. An organic layer can be formed using a vacuum depositionprocess or an application process, for example. Examples of an inorganicphotoelectric conversion film include a quantum dot inorganicphotoelectric conversion film that uses a quantum dot filmy layercontaining fine semiconductor crystals in place of an organic layer, anda perovskite-type inorganic photoelectric conversion film including aphotoelectric conversion layer including a transition metal oxide havinga perovskite structure.

The display unit 103 includes a plurality of light emitting elements.Examples of the light emitting element include a liquid crystal display(LCD), an inorganic light emitting diode (LED), an organic LED (OLED),and a quantum dot LED (QLED). Examples of material used for an inorganicLED include aluminum, gallium, arsenicum, phosphorus, indium, nitrogen,selenium, zinc, diamond, zinc oxide, and a perovskite semiconductor. Bymaking a PN junction structure using these materials, light havingenergy (wavelength) equivalent to a bandgap of materials is emitted. Forexample, an organic LED may include a light emitting layer containing atleast one type of organic light emitting material between a pair ofelectrodes, may include a plurality of light emitting layers, may have astructure in which a plurality of organic layers is stacked, may includea light emitting layer made of single material, or may include a lightemitting layer made of a plurality of materials. Light from a lightemitting layer may be fluorescence or phosphorescence, or may besingle-color light emission (blue, green, red, etc.) or white lightemission. In addition, an organic layer can be formed using a vacuumdeposition process or an application process, for example.

The pupil detection unit 104 includes a plurality of light receivingelements. Examples of the light receiving element include aphotoelectric conversion element for obtaining image information thathas been described above in the above-described imaging unit, and adistance measuring sensor for acquiring distance information from apupil. As a system of the distance measuring sensor, a Time-Of-Flight(TOF) system can be used, but an element that can acquire vectorinformation including another type of distance information may be used.

The imaging display device may have a structure in which at least fourchips of the imaging unit 101, the processing unit 102, the display unit103, and the pupil detection unit 104 are stacked and the chips areelectrically connected with each other by a semiconductor process. Theconfigurations of the imaging unit 101, the processing unit 102, thedisplay unit 103, and the pupil detection unit 104 can be appropriatelychanged.

A second exemplary embodiment will be described with reference to FIGS.4A, 4B, 4C, 4D, and 4E. FIGS. 4A to 4E are schematic diagramsillustrating an imaging display device according to the presentexemplary embodiment. Other configurations are similar to those in thefirst exemplary embodiment. In the present exemplary embodiment, thepupil detection unit 104 further includes a function of detecting amovement of a line of sight and a state of a pupil. In addition to thepupil detection unit 1041, the pupil detection unit 104 includes a pupildetection unit 1042 for obtaining pupil information as imageinformation. In the present exemplary embodiment, pupil information fordetecting a line of sight and the state of a pupil is acquired fromimage information on the pupil. In the processing unit 102, pupil-basedadjustment processing is performed based on information from the pupildetection unit 1041 and the pupil detection unit 1042, and generatedpupil-based adjusted image information is displayed on the display unit103. Because the movement of the line of sight and the state of thepupil can be thereby detected and adjusted, an image corresponding tothe line of sight is displayed on the display unit 103. Specifically,for example, a region other than a region of interest (will also bereferred to as an ROI region) can be displayed at low resolution, or animage in which a region of interest is enlarged or reduced can bedisplayed. A region of interest is estimated by the processing unit 102based on a line of sight tracking result. In addition, pupil-basedadjusted image information having luminance adjusted in accordance withthe state of the pupil such as a size of the pupil, and being adjustedto a real image can also be generated. Accordingly, an image directlyviewed by an eye and an image displayed on the imaging display devicebecome consistent with each other. Moreover, an image corresponding to aline of sight is displayed. Thus, an image less uncomfortable for theuser, that is to say, a display image similar to an image directlyviewed by an eye can be obtained.

Aside from a method of acquiring an image of a pupil, the pupildetection unit 1042 can employ a method of detecting an outer rim of aniris of an eye, or a method of identifying the position of a pupil byemitting infrared light and using corneal reflection. The pupildetection unit 1042 can apply an arbitrary method in eye tracking.

The pupil detection unit 104 of the imaging display device according tothe present exemplary embodiment will now be described with reference toFIGS. 4A, 4B, 4C, 4D, and 4E. FIGS. 4A, 4B, 4C, 4D, and 4E are diagramscorresponding to FIGS. 2A, 2B, 2C, and 2D. The imaging unit 101 and thepupil detection unit 1041 are similar to those in the first exemplaryembodiment. As long as the pupil detection unit 1042 is at a position atwhich an image including information regarding the pupil and the stateof the pupil can be captured, the pupil detection unit 1042 may be onthe central axis C1 of the pupil or may not be on the central axis C1.

In FIG. 4A, the imaging unit 101 and the pupil detection unit 1041 areon the central axis C1 of the pupil, similar to FIG. 2A. In addition,the pupil detection unit 1042 is on the central axis C1. In other words,the pupil detection unit 1041 and the pupil detection unit 1042 can beregarded as being at the same position. In such a case, pupil-basedadjusted image information can be generated using the vector informationP1, similar to FIG. 2A. Vector information between the pupil detectionunit 1042 and the pupil detection unit 1041 can be preliminarily held bythe processing unit 102 in the manufacturing of the imaging displaydevice.

In FIG. 4B, the imaging unit 101 and the pupil detection unit 1041 arenot on the central axis C1 of the pupil, similar to FIG. 2B. The pupildetection unit 1041 is on the central axis C2 of the imaging unit 101,and the central axis C1 and the central axis C2 are offset. In addition,the pupil detection unit 1042 is positioned on the central axis C2. Inother words, the pupil detection unit 1041 and the pupil detection unit1042 can be regarded as being at the same position. In such a case,pupil-based adjusted image information can be also generated similarlyto FIG. 2B.

In FIG. 4C, similar to FIG. 2C, while the pupil detection unit 1041 isarranged on the central axis C1 of the pupil, the imaging unit 101 isnot arranged on the central axis C1 of the pupil. The imaging unit 101has the central axis C2 offset from the central axis C1. The pupildetection unit 1042 is on the central axis C1. In other words, the pupildetection unit 1041 and the pupil detection unit 1042 can be regarded asbeing at the same position. In such a case, pupil-based adjusted imageinformation can be generated similarly to FIG. 2C.

In FIG. 4D, similar to FIG. 2B, the imaging unit 101 and the pupildetection unit 1041 are not on the central axis C1 of the pupil. Thepupil detection unit 1041 is on the central axis C2 of the imaging unit101, and the central axis C1 and the central axis C2 are offset. Inaddition, the pupil detection unit 1042 is on the central axis C1. Insuch a case, pupil-based adjusted image information can be generatedsimilarly to FIG. 2B. In this case, pupil information from the pupildetection unit 1042 can be adjusted using vector information P3 betweenthe pupil detection unit 1041 and the pupil detection unit 1042.

In FIG. 4E, similar to FIG. 2C, while the pupil detection unit 1041 isarranged on the central axis C1 of the pupil, the imaging unit 101 isnot arranged on the central axis C1 of the pupil. The imaging unit 101has the central axis C2 offset from the central axis C1. The pupildetection unit 1042 is arranged on neither the central axis C1 nor thecentral axis C2. The pupil detection unit 1042 has a central axis C3that is offset from the central axis C1 and is offset from the centralaxis C2. The processing unit 102 preliminarily includes the vectorinformation P2 between the pupil detection unit 1041 and the imagingunit 101, and the vector information P3 between the pupil detection unit1042 and the pupil detection unit 1041. The pupil detection unit 1041acquires the vector information P1 from the pupil. Based on the piecesof vector information P1 to P3, the processing unit 102 adjusts imageinformation acquired by the imaging unit 101, and acquires pupil-basedadjusted image information.

In addition, pupil information may be acquired using a plurality ofpupil detection units 1042.

An imaging display device according to a third exemplary embodiment willbe described with reference to FIGS. 5A to 5C. FIGS. 5A to 5C illustratemodified examples of the imaging display devices according to the firstexemplary embodiment that are illustrated in FIGS. 1A to 1C.

FIG. 5A is a schematic diagram illustrating an imaging display device200 according to the present exemplary embodiment. The imaging displaydevice 200 is different from the imaging display device 100 according tothe first exemplary embodiment that is illustrated in FIG. 1A in thatthe processing unit 102 further includes an AI unit 107 equipped with anintelligence (hereinafter, abbreviated as “AI”) unit. The AI unit 107may include a deep learning function. With this configuration, whenpupil-based adjustment processing is performed based on informationacquired by the pupil detection unit 104, the processing unit 102 canenhance the accuracy of pupil-based adjusted processing and increase thespeed of pupil-based adjusted processing by using the deep learningfunction of the AI unit 107.

For example, by learning together with environmental information such astemperature humidity information, acceleration information, and pressureinformation, more accurate pupil-based adjusted image information can begenerated. In addition, pupil-based adjusted image information can begenerated more quickly by learning an action pattern from past pupilinformation on the user. By enhancing the accuracy of pupil-basedadjusted processing, a difference between information directly viewed byan eye and information displayed on the imaging display device becomessmaller to each other, and the user can use the imaging display devicecomfortably. In addition, by increasing the speed of pupil-basedadjusted processing, a time from when image information is acquired towhen the image information is displayed can be shortened, and latencycan be made smaller. The functions of the AI unit 107 are not limited tothe above described functions. The functions of the AI unit 107 are notspecifically designated as long as the functions enhance the performanceof the imaging display device.

FIG. 5B illustrates a modified example of the imaging display device 120illustrated in FIG. 1B. The processing unit 102 of an imaging displaydevice 220 communicates with the processing device 105. The processingunit 102 and the processing device 105 connect with each other via anetwork. The processing device 105 is disposed on the outside of theimaging display device 220, and may be on a cloud, for example. In theimaging display device 220, not the processing unit 102 but theprocessing device 105 includes the AI unit 107. The processing unit 102and the processing device 105 exchange information with each other, andgenerate pupil-based adjusted image information from image informationand pupil information. In FIG. 5B, image information and pupilinformation respectively acquired by the imaging unit 101 and the pupildetection unit 104 are converted into pupil-based adjusted imageinformation by the processing unit 102 that has obtained informationfrom the processing device 105. In this manner, the imaging displaydevice 220 can generate pupil-based adjusted image information usinginformation accumulated in an external device.

FIG. 5C is a schematic diagram illustrating a modified example of theimaging display device 130 illustrated in FIG. 1C. The processing unit102 of an imaging display device 230 communicates with the processingdevice 106, and the processing device 106 further communicates withanother processing device 105. The processing device 106 includes the AIunit 107. The processing device 106 is on a cloud and performs dataaccumulation, for example. The processing device 105 is providedseparately from the imaging display device 230 and the processing device106. The processing unit 102 and the processing device 105 connect witheach other via a network, and the processing device 106 and theprocessing device 105 connect with each other via a network. In FIG. 5C,the processing unit 102 receives setting information accumulated in theprocessing device 106, and generates pupil-based adjusted imageinformation based on the setting information. The setting informationincludes basic information on an environment and a target object, andvarious values for generating pupil-based adjusted image information.The processing unit 102 also transmits a plurality of pieces ofinformation including image information and pupil information from theimaging unit 101 and the pupil detection unit 104, to the processingdevice 106. The plurality of pieces of information is transmitted to theprocessing device 105 via the processing device 106. Based on theplurality of pieces of received information, the processing device 105generates various values for generating pupil-based adjusted imageinformation, and transmits the generated various values to theprocessing device 106. The processing device 106 updates basicinformation and various values that are accumulated therein, and holdsthe updated basic information and values as new information. In thismanner, the imaging display device 230 can generate pupil-based adjustedimage information using information accumulated in an external device.

In FIG. 5A, the processing unit 102 transmits pupil-based adjusted imageinformation to the display unit 103 based on the image information andthe pupil information respectively obtained by the imaging unit 101 andthe pupil detection unit 104. The processing unit 102 can process notonly the image information and the pupil information but also othertypes of information such as temperature humidity information,acceleration information, and pressure information. The processing unit102 illustrated in FIG. 5B, and the processing unit 102, the processingdevice 105, and the processing device 106 that are illustrated in FIG.5C are similar to the case illustrated in FIG. 5A.

In the case of using the imaging display device according to the presentexemplary embodiment as a wearable device, a smaller processing dataamount in a processing unit is more desirable. This is because it isnecessary to make a wearable terminal lightweight and thin as far aspossible, and a chip of a processing unit can be made smaller withdecrease in a load on data processing. As a method of reducing a load ofa data processing amount, for example, there is a method of performingAI processing in a separate device (cloud, etc.) as illustrated in FIGS.5B and 5C. In addition, as a method of reducing a processing amount,there are a method of decreasing resolution of a portion other than aregion of interest, a method of making a portion other than a region ofinterest a still image, and a method of performing not color processingbut monochrome processing on a portion other than a region of interest.

In a fourth exemplary embodiment, adjustment including not only pupilinformation but also prediction is performed on image informationobtained by image capturing by the imaging unit 101. In the adjustmentincluding prediction, the processing unit 102 generates prediction imageinformation that predicts future, from image information acquired by theimaging unit 101, simultaneously with pupil-based adjustment. Theprediction image information is displayed on the display unit 103. Acharacteristic point of the processing unit 102 lies in that theprocessing unit 102 includes not only a function of performingpupil-based adjustment processing based on image information obtained byimage capturing by the imaging unit 101 and information acquired by thepupil detection unit 104, but also a function of generating predictionimage information that predicts future. With this configuration, notonly pupil-based adjusted image information that is based on theposition of a pupil is displayed, but also a temporal difference fromwhen image information is acquired to when the image information isdisplayed, that is to say, latency can be reduced. Thus, for example,when the user performs an operation of catching a moving object, theuser can desirably use the imaging display device. In this case, theprocessing unit 102 may include the AI unit 107 as illustrated in FIG.5A. Furthermore, the AI unit 107 may perform adjustment for enhancingthe performance of the imaging display device.

An operation of generating prediction image information that predictsfuture will be described with reference to FIGS. 6A and 6B. FIGS. 6A and6B are diagrams illustrating an operation of the imaging display deviceaccording to the present exemplary embodiment, and are diagramsillustrating a relationship between image information on one frame at acertain time and prediction image information. In FIGS. 6A and 6B, imageinformation at a time Tn is denoted by An, and a future imageinformation (prediction image information) processed by the processingunit 102 is denoted by Bn.

An operation of the imaging display device according to the presentexemplary embodiment will be described with reference to FIG. 6A. Inthis operation, the imaging unit 101 performs an image capturingoperation of obtaining image information A⁻², at a time T⁻², imageinformation A⁻¹ at a time T⁻¹, image information A₀ at a time T₀, andimage information A₊₁ at a time T₊₁. Next, based on the pieces of inputimage information A⁻¹, A₀, and A₊₁, the processing unit 102 generatespieces of prediction image information B₀, B₊₁, and B₊₂. Then, theprocessing unit 102 outputs the pieces of prediction image informationB₀, B₊₁, and B₊₂ to the display unit 103. The display unit 103 performsa display operation of displaying an image that is based on theprediction image information B₀, at the time T₀, an image that is basedon the prediction image information B₊₁, at the time T₊₁, and an imagethat is based on the prediction image information B₊₂, at the time T₊₂.

In other words, the imaging unit 101 performs an image capturingoperation of obtaining the image information A⁻¹ at the certain timeT⁻¹, and performs an image capturing operation of obtaining the imageinformation A₀ that is different from the image information A⁻¹, at thetime T₀ later than the certain time T⁻¹. At the time T₀, the displayunit 103 performs a display operation of displaying an imagecorresponding to the prediction image information B₀ generated from theimage information A⁻¹. Furthermore, at the time T₀ later than the timeT₀, the imaging unit 101 performs an image capturing operation ofobtaining the image information A₊₁ that is different from the imageinformation A₀. Then, the display unit 103 performs a display operationof displaying an image corresponding to the prediction image informationB₊₁ generated from the image information A₀.

A display timing of prediction image information according to thepresent exemplary embodiment will be described. At a certain time, theprocessing unit 102 according to the present exemplary embodimentgenerates prediction image information in such a manner as to reduce alag between image information obtained by performing image capturing bythe imaging unit 101, and an image displayed by the display unit 103. Itis desirable to set a display timing of the prediction image informationin the following manner.

First of all, at an arbitrary time Tn, the imaging unit 101 captures animage. A time at which prediction image information at the time Tn isgenerated by the display unit 103 and an image that is based on theprediction image information at the time Tn is displayed by the displayunit 103 is denoted by Tm. In this case, a difference ΔT between animage capturing timing and a display timing can be represented by (1):

ΔT=Tn−Tm  (1).

In this case, a display frame rate (frame per second (fps)) being thenumber of images to be displayed by the display unit 103 per second isdenoted by DFR. The imaging display device is controlled in such amanner that the difference ΔT satisfies Inequality (2). Morespecifically, the imaging display device is controlled in such a mannerthat the difference ΔT satisfies Inequality (3).

−2/DFR≤ΔT≤2/DFR  (2)

−1/DFR≤ΔT≤1/DFR  (3)

For example, when a display frame rate is 240 (fps), a time taken forone image (one frame) from when the image is captured to when the imageis displayed is about 4×10⁻³ (seconds). Accordingly, the difference ΔTcan be calculated as follows:

−4×10⁻³ ≤ΔT≤4×10⁻³  (4).

By displaying, at such a timing, an image that is based on predictionimage information, a moving image with a less lag between a real imageand a displayed image can be displayed. The above-described moving imagedisplay can also be said to be real-time display. Accordingly, in thepresent exemplary embodiment, real-time display, strictly speaking,pseudo real-time display can be performed. While the present exemplaryembodiment can also be applied to a still image, it is effective toperform the operation on a moving image.

Aside from displaying prediction image information at such a timing,such a time lag can also be utilized when prediction image informationis generated. Image information obtained by image capturing by theimaging unit 101 at an arbitrary time is denoted by An. Imageinformation displayed by the display unit 103 at the same time isdenoted by Dn. In this case, a difference between the pieces of imageinformation, that is to say, a temporal shift amount can be representedas ΔA=Dn−An. In the exemplary embodiment illustrated in FIG. 6A, Dn=Bnis obtained. In other words, a temporal difference between imageinformation obtained by image capturing by the imaging unit 101 at acertain time, that is to say, a real event (real image) at the certaintime and image information displayed by the display unit 103 satisfies±4×10⁻³ (seconds). A temporal difference between pieces of imageinformation being ±4×10⁻³ (seconds) means that an image displayed by thedisplay unit 103 is an image delayed by 4×10⁻³ (seconds) from a realimage at the certain time or an image brought forward by 4×10⁻³(seconds). It is desirable that prediction image information isgenerated under such a condition. The comparison between the imageinformation An and the image information Dn can be performed using RAWdata of the image information An and the image information Dn, forexample. Then, the image information Dn is obtained in a manner suchthat the image information Dn is within ±4×10⁻³ (seconds) when aroot-mean-square of the difference is calculated. Using informationregarding the difference, the processing unit 102 sets the followingvarious parameters for generating prediction image information.

Especially in the case of performing additional image processing, a lagbecomes 100×10⁻³ (seconds). However, by generating prediction imageinformation according to the present exemplary embodiment, an imagewithout a temporal difference from a real image can be displayed.

Examples of the additional image processing include dark field of viewimage processing of increasing luminance of a dark image, enlargementimage processing of displaying a small subject in an enlarged size, andtemperature display processing of displaying temperature in an image. Bythe operation according to the present exemplary embodiment, real-timedisplay can be performed even in a case where a time for performing suchimage processing is added.

Next, an operation in FIG. 6B will be described. In this operation, theimaging unit 101 performs an image capturing operation of obtainingimage information A⁻², at a time T⁻², image information A⁻¹ at a timeT⁻¹, image information A₀ at a time T₀, and image information A₊₁ at atime T₊₁. Next, based on the pieces of input image information A⁻¹, A₀,and A₊₁, the processing unit 102 generates pieces of prediction imageinformation B₊₁, B₊₂, and B₊₃. Then, the processing unit 102 outputs thepieces of prediction image information B⁻⁰, B₊₂, and B₊₃ to the displayunit 103. The display unit 103 performs a display operation ofdisplaying an image that is based on the prediction image informationB₊₁, at the time T₀, an image that is based on the prediction imageinformation B₊₂, at the time T₊₁, and an image that is based on theprediction image information B₊₃, at the time T₊₂. In other words, imageinformation to be obtained by performing image capturing at a time T₀ ispredicted and displayed at the time T₀. In this manner, information at atime forward of an image capturing time can be displayed at the imagecapturing time. By continuously repeating the operation, an imageforward of a real image can be continuously displayed. That is to say,an image can be displayed as a video.

Source image information on prediction image information will bedescribed. For example, in the description of FIG. 6A, the predictionimage information B₀ is generated based on the image information A⁻¹. Inthe description of FIG. 6B, the prediction image information B⁻¹ isgenerated based on the image information A⁻¹. In other words, one pieceof prediction image information is generated based on one piece of imageinformation. Alternatively, one piece of prediction image informationmay be generated based on two or more pieces of image information. Forexample, in FIG. 6A, the prediction image information B₀ may begenerated based on the pieces of image information A⁻², and A⁻¹. In FIG.6B, the prediction image information B⁻¹ may be generated based on thepieces of image information A⁻² and A⁻¹. Accordingly, prediction imageinformation can be generated using at least one piece of imageinformation.

A frame rate in the present exemplary embodiment will be described.First of all, the number of pieces of image information to be acquiredby the imaging unit 101 per second will be referred to as an imagecapturing frame rate SFR (fps). In addition, as described above, thenumber of pieces of image information to be displayed by the displayunit 103 per second will be referred to as a display frame rate DFR(fps). In this case, a relationship between frame rates in FIGS. 6A and6B in the present exemplary embodiment is represented as SFR=DFR.Alternatively, an image capturing frame rate and a display frame ratemay be different. In particular, it is desirable that SFR≥DFR isobtained. This is because prediction image information can be generatedfrom a plurality of pieces of image information obtained by imagecapturing.

By the imaging display device according to the present exemplaryembodiment, it is possible to provide an imaging display device withwhich uncomfortable feeling for the user is reduced.

An imaging display device according to a fifth exemplary embodiment willbe described with reference to FIGS. 7A, 7B, 8A, 8B, and 8C. FIG. 7Aillustrates an imaging display device 720 corresponding to FIG. 1B. FIG.7B is a block diagram illustrating the imaging unit 101. In the presentexemplary embodiment, the imaging unit 101 includes n (n is a naturalnumber) imaging units 101 (1) to 101 (n). The pupil detection unit 104can detect a pupil including line of sight information. The pupildetection unit 104 may acquire an image information including line ofsight information. The processing unit 102 can acquire line of sightinformation detected by the pupil detection unit 104, and changeoperations of the plurality of imaging units 101. Examples of theplurality of imaging units 101 include an imaging apparatus including aphotodiode as described in the first exemplary embodiment.Alternatively, the plurality of imaging units 101 may be a photoncounting sensor, such as a single-photon avalanche diode (SPAD). TheSPAD is an imaging apparatus including an avalanche diode. In this case,clear image information can be displayed even under an environment withlow luminance.

FIG. 8A is a cross-sectional schematic view of the imaging displaydevice 720 according to the present exemplary embodiment. The imagingdisplay device 720 includes a plurality of imaging units 101. In FIG.8A, the imaging display device 720 includes three imaging units 101 (1)to 101 (3). The pupil detection unit 104 detects a line of sight. When aline of sight exists near the imaging unit 101 (2), the processing unit102 can perform the following operation: the processing unit 102generates pupil-based adjusted image information using only imageinformation acquired by the imaging unit 101 (2) among the plurality ofimaging units 101. The processing unit 102 does not use imageinformation acquired by the other imaging units 101 (1) and 101 (3).Alternatively, the processing unit 102 can generate pupil-based adjustedimage information using all image information acquired by the imagingunit 101 (2), and using image information acquired by the other imagingunits 101 (1) and 101 (3) with a reduced information amount. As areduction method of an information amount of image information, thereare a method of using only luminance information, a method of decreasingthe resolution of an image by thinning out pixels, and a method of usingan average value or a median value of a plurality of pixels.Furthermore, as a reduction method, a method of using a reduced numberof output bits from an imaging unit of a pixel signal value, and amethod of using only information to be used for distance measurement orfocusing can also be used. The reduction of an information amount ofimage information may be performed in the imaging unit 101 or may beperformed by the processing unit 102. When the processing unit 102 usesonly image information acquired by the imaging unit 101 (2) existingnear a line of sight, the other imaging units 101 (1) and 101 (3) may beshifted to a power saving state by changing the imaging units 101 (1)and 101 (3) into a sleep mode or stopping power supply to the imagingunits 101 (1) and 101 (3). These types of control can be performed bythe processing unit 102. FIG. 8B illustrates a state in which the lineof sight S1 moves to the imaging unit 101 (3). According to a positionof the line of sight, the processing unit 102 uses image informationacquired by the imaging unit 101 (3).

FIG. 8C illustrates a modified example of FIG. 8A. In FIG. 8C, theimaging unit 101 can move to a desired position based on line of sightinformation detected by the pupil detection unit 104. In FIG. 8C, aplurality of image sensors is arranged as the imaging units 101. Amongthese image sensors, the imaging unit 101 (2) is arranged on a line ofsight S1. When the line of sight moves, the pupil detection unit 104detects the movement and the imaging unit 101 moves in such a mannerthat the imaging unit 101 (2) is arranged on the line of sight. Themovement can be performed by a power unit disposed in a casing of theimaging display device. With this configuration, as compared with thecase illustrated in FIG. 8A, while the imaging unit 101 is moved, anoperation of switching roles of a plurality of image sensors isunnecessary. Furthermore, since a positional relationship between a lineof sight and each imaging unit is defined to one, a load on subsequentimage processing can be reduced.

In the imaging display device according to the present exemplaryembodiment, when a surface on which a plurality of imaging units 101 isarranged is curved, vector information between a pupil and the imagingunit 101 is fixed, and therefore a load on subsequent image processingcan be further reduced, which is more desirable.

An imaging display device according to a sixth exemplary embodiment candisplay an image that uses light other than visible light (near-infraredlight, infrared light, ultraviolet light, etc.). For example, theimaging unit 101 includes a photoelectric conversion element that candetect a visible light region, and a photoelectric conversion elementthat can detect light in a waveband other than the visible light region.For example, the imaging unit 101 includes at least two imagingapparatuses. One of the imaging apparatuses is an imaging apparatusequipped with a photoelectric conversion element for visible light, andthe other one imaging apparatus is an imaging apparatus equipped with aphotoelectric conversion element for light other than visible light.Alternatively, the imaging unit 101 includes one imaging apparatus. Theone imaging apparatus may include at least one photoelectric conversionelement for visible light, and at least one photoelectric conversionelement for light other than visible light.

By such an imaging unit 101, in addition to image information in avisible light region, an image signal in a region other than the visiblelight region including a near-infrared light region can also beacquired. Using these pieces of image information, the processing unit102 generates pupil-based adjusted image information in one visiblelight region. More specifically, the processing unit 102 generatespupil-based adjusted image information by adjusting image informationusing pupil information and information in the near-infrared lightregion. With this configuration, even in a situation in whichsensitivity of a visible light region is low, an image with enhancedsensitivity is displayed. In other words, according to the imagingdisplay device according to the present exemplary embodiment, an imageinvisible to human eyes can also be displayed. Such an imaging displaydevice according to the present exemplary embodiment can also be appliedto a night vision device, a surveillance device, binocular glasses, atelescope, and a medical detection device, for example.

An imaging display device according to a seventh exemplary embodimentwill be described with reference to FIGS. 9A and 9B. FIGS. 9A and 9B areschematic diagrams illustrating the imaging display device 100 thatcorrespond to FIGS. 2A, 2B, 2C, and 2D. In FIGS. 2A, 2B, 2C, and 2D, thecentral axis C1 of the pupil and the center of the display unit 103coincide with each other. In the present exemplary embodiment, thedescription will be given of a case where the central axis C1 of thepupil and a central axis C4 of the display unit 103 do not coincide witheach other. Since the other configurations in FIGS. 9A and 9B aresimilar to those in FIGS. 2A, 2B, 2C, and 2D, the detailed descriptionwill be omitted.

In FIG. 9A, the central axis C4 of the display unit 103 is arranged at adistance from the central axis C1 of the pupil by a predetermineddistance in the X direction. The imaging unit 101 and the pupildetection unit 1041 are arranged on the central axis C4 of the displayunit 103. The pupil detection unit 1041 acquires the vector informationP1 between the pupil and the pupil detection unit 104. Based on thevector information P1, image information acquired by the imaging unit101 is adjusted together with the positions of the display unit 103 andthe pupil. By using such pupil-based adjusted image information, thedisplay unit 103 can display an image of which the center position ischanged in accordance with the position of the pupil. By theabove-described processing, a difference from a real image can befurther reduced.

In FIG. 9B, similarly to FIG. 9A, the central axis C4 of the displayunit 103 is arranged at a distance from the central axis C1 of the pupilby a predetermined distance in the X direction. The imaging unit 101 isarranged on the central axis C4 of the display unit 103. The pupildetection unit 1041 is arranged on the central axis C1 of the pupil. Thepupil detection unit 1041 acquires the vector information P1 between thepupil and the pupil detection unit 104. In addition, the processing unit102 holds the vector information P2 between the pupil detection unit1041 and the imaging unit 101. Image information acquired by the imagingunit 101 is adjusted using the vector information P1 and the vectorinformation P2. By using such pupil-based adjusted image information,the display unit 103 can display an image of which the center positionis changed in accordance with the position of the pupil. By theabove-described processing, a difference from a real image can befurther reduced.

An example of applying the imaging display device according to eachexemplary embodiment to a wearable device will be described withreference to FIG. 10. The imaging display device can be applied to awearable device such as smart glasses, a head mounted display (HMD), andsmart contact lenses, for example.

FIG. 10 is a schematic diagram illustrating smart glasses 1000. Thesmart glasses 1000 will also be referred to as an eyewear-shaped imagingdisplay device or glasses. The smart glasses 1000 include aneyewear-shaped casing. The casing will also be referred to as a frame.The frame is provided with the imaging display device according to eachexemplary embodiment. Specifically, the smart glasses 1000 at leastinclude an imaging unit 1001, a processing unit 1002, a display unit1003, and a pupil detection unit 1004. Two imaging units 1001 areprovided on the frame side surfaces of the glasses. Alternatively, theimaging units 1001 may be provided on lenses. The processing units 1002are stored in temples of the glasses. The display unit 1003 is providedat an arbitrary position depending on the display format, and may beincluded in lenses 1011. In any case, the display unit 1003 displays animage on the lenses 1011. The pupil detection unit 1004 is stored on thepupil side at the center of the two glasses lenses, and may be providedon the lens or the frame side surface. The processing unit 1002 mayinclude an AI unit. The smart glasses 1000 may include an externalinterface and the processing unit 1002 may communicate with an externalAI unit. The frame may include a power source unit and may include aninterface unit for performing wireless connection with the outside.

The smart glasses 1000 illustrated in FIG. 10 may include two imagingdisplay devices for a left eye and a right eye. In this case, in theimaging display devices for the left eye and the right eye, an imagecapturing timing and a display timing can be arbitrarily set.Specifically, an operation of performing image capturing at the sametime and displaying an image at a different time, or an operation ofperforming image capturing at a different time and displaying an imageat the same time can be performed.

The imaging unit 1001 and the display unit 1003 may be disposed atdifferent positions as illustrated in FIG. 10. Alternatively, theimaging unit 1001, the display unit 1003, and the pupil detection unit1004 may be stacked on a line of sight.

An imaging display device according to a ninth exemplary embodiment willbe described with reference to FIGS. 11A, 11B, and 11C. FIG. 11A is aschematic diagram illustrating an imaging display device 810 accordingto the present exemplary embodiment. FIG. 11A is a schematic diagramcorresponding to FIG. 7A. In FIG. 11A, the same configurations as thosein the other exemplary embodiments are assigned the same referencenumerals and the redundant description will be omitted. In addition, thedescription of configurations and operations that are similar to thosein the other exemplary embodiments will be omitted. In the presentexemplary embodiment, the description will be given of a technique thatcan reduce a difference between a real event and an image displayed onan imaging display device, by factoring in a positional relationshipbetween an imaging unit and a pupil.

In FIG. 11A, the pupil detection unit 104 can acquire a pupil includingline of sight information, as image information. The processing unit 102can acquire line of sight information detected by the pupil detectionunit 104, set a weight to a plurality of imaging units 101, and generateone piece of image information. Specifically, based on the line of sightinformation detected by the pupil detection unit 104, an imaging unit101 serving as a main imaging unit can be set from among a plurality ofimaging units 101, and one piece of image information can be generatedin a manner such that image information acquired by the main imagingunit 101 is interpolated using image information acquired by an imagingunit 101 other than the main imaging unit 101. Thus, an image includinga positional relationship between an imaging unit and a pupil can begenerated. A difference between a displayed image and a real event canbe therefore reduced. By the technique according to the presentexemplary embodiment, wide-range external information can be displayedon a display unit. The wide range includes a field angle and a distance.

An operation according to the present exemplary embodiment will bedescribed with reference to FIGS. 11B and 11C. FIG. 11C is across-sectional schematic view of the imaging display device 810. Theimaging display device 810 includes a plurality of imaging units 101.Each of the imaging units 101 can be an image sensor including aphotodiode, for example. The imaging display device 810 includes atleast two imaging units. Specifically, the imaging display device 810includes an imaging unit 101 (4) and an imaging unit 101 (5). The pupildetection unit 104 detects a line of sight.

FIG. 11B is an operation flow diagram of the imaging display device 810.First of all, the imaging unit 101 (4) and the imaging unit 101 (5) eachacquire image information in steps 1101, 1102 and 1103. The pupildetection unit 104 detects the position of a pupil and pupilinformation. As pupil information, line of sight information isincluded. For example, the pupil detection unit 104 detects toward whichimaging unit a line of sight is inclined with respect to a pupil centralaxis. When the line of sight is inclined toward the imaging unit 101 (4)with respect to the pupil central axis, the processing unit 102 performsthe following operation.

First of all, the processing unit 102 generates pupil-based adjustedimage information from image information acquired by a plurality ofimaging units 101. Based on line of sight information, the processingunit 102 sets, as main image information, image information acquired bythe imaging unit 101 (4) among the plurality of imaging units in step1105 or 1106. The processing unit 102 generates one piece of imageinformation in a manner such that the main image information isinterpolated using image information acquired by the imaging unit 101(5) other than the main imaging unit 101 (4) in step 1107. Theprocessing unit 102 sets the one piece of image information to theimaging units 101 (4) and 101 (5). By such processing, a natural displayimage in which a positional relationship between an imaging unit and apupil, for example, a positional relationship between an imaging unitand a line of sight, is factored in can be generated. A differencebetween a real event and an image displayed on an imaging display devicecan be therefore reduced. In addition, by an AI unit disposed in theprocessing unit 102 and using deep learning, accuracy and a processingspeed in generating one piece of image information from a plurality ofpieces of image information can be increased.

In the present exemplary embodiment, the description has been given ofprocessing performed after image information is acquired from theimaging unit 101 (4) and the imaging unit 101 (5). Alternatively, aftera line of sight is detected, the imaging unit 101 (4) serving as a mainimaging unit may be selected and the imaging unit 101 (5) serving as asub imaging unit may be selected. Then, the main imaging unit 101 (4)and the sub imaging unit 101 (5) may acquire image information, and theprocessing unit 102 may perform the above-described processing.

Another operation as illustrated in FIG. 12 may be performed. FIG. 12 isanother operation flow diagram according to the present exemplaryembodiment that corresponds to FIG. 11B. The pupil detection unit 104detects a line of sight in step 1203. After a line of sight is detected,the imaging unit 101 (4) serving as a main imaging unit and the imagingunit 101 (5) serving as a sub imaging unit are selected in accordancewith the position of the line of sight in step 1204. Next, an imagingcondition of each imaging unit is set in steps 1205 and 1206. Forexample, the imaging unit 101 (4) is set to high resolution imagecapturing as a main imaging unit, and the imaging unit 101 (5) is set tolow resolution image capturing as a sub imaging unit. After that, instep 1207, one piece of image information from two pieces of imageinformation is generated, whereby a display image factoring in apositional relationship between an imaging unit and a line of sight canbe generated.

An imaging display device according to a tenth exemplary embodiment willbe described with reference to FIGS. 13A, 13B, and 13C. FIG. 13A is aschematic diagram illustrating an imaging display device 910 accordingto the present exemplary embodiment. FIG. 13A is a schematic diagramcorresponding to FIG. 7A. In FIG. 13A, the same configurations as thosein the other exemplary embodiments are assigned the same referencenumerals and the redundant description will be omitted. In addition, thedescription of configurations and operations that are similar to thosein the other exemplary embodiments will be omitted. In the presentexemplary embodiment, the description will be given of a technique bywhich a difference between a real event and an image displayed on animaging display device can be reduced by factoring in a positionalrelationship between an imaging unit and a pupil.

In FIG. 13A, the pupil detection unit 104 can detect a pupil includingline of sight information, as image information. The processing unit 102can acquire line of sight information detected by the pupil detectionunit 104, set a weight to a plurality of imaging units 101, and generateone piece of image information. Specifically, the processing unit 102sets a line of sight region from the line of sight information detectedby the pupil detection unit 104. Based on the line of sight region, fromamong the plurality of imaging units 101, the processing unit 102selects an imaging unit 101 for creating image information correspondingto the line of sight region, and another imaging unit 101 that canacquire widest-range external information. Then, the processing unit 102generates at least these pieces of image information as one piece ofimage information. By using the imaging display device 910, at least onetype of display can be performed. As one type of display, wide-rangeexternal information can be displayed on a display unit. As another typeof display, the line of sight region can be displayed at high resolutionand other regions can be displayed at low resolution. By the imagingdisplay device 910, a natural display image can be displayed and a loadon image processing can also be reduced.

An operation according to the present exemplary embodiment will bedescribed with reference to FIGS. 13B and 13C. FIG. 13C is across-sectional schematic view of the imaging display device 910according to the present exemplary embodiment. The imaging displaydevice 910 includes a plurality of imaging units 101. Each of theimaging units 101 can be an image sensor including a photodiode, forexample. The imaging display device 910 includes at least two imagingunits. Specifically, the imaging display device 910 includes an imagingunit 101 (6) and an imaging unit 101 (7). The pupil detection unit 104detects a line of sight region.

FIG. 13B is an operation flow diagram of the imaging display device 910according to the present exemplary embodiment. First of all, the imagingunit 101 (6) and the imaging unit 101 (7) each acquire image informationin steps 1301 and 1302. The pupil detection unit 104 detects a line ofsight region in step 1303. The line of sight region means a region towhich a line of sight is oriented. In this case, the line of sight isoriented toward the imaging unit 101 (6). Based on the obtained line ofsight region, the processing unit 102 performs the following operation.

The processing unit 102 generates pupil-based adjusted image informationfrom image information acquired by the plurality of imaging units 101 instep 1304. In addition, the processing unit 102 extracts imageinformation corresponding to the line of sight region, from the imageinformation acquired by the imaging unit 101 (6) and the line of sightinformation acquired by the pupil detection unit 104 in step 1305.Meanwhile, the processing unit 102 generates one piece of imageinformation by combining the image information acquired by the imagingunit 101 (7) with the image information corresponding to the line ofsight region that has been extracted from information from the imagingunit 101 (6) in step 1306. By such processing, a display image factoringin a line of sight can be generated. A difference between a real eventand an image displayed on an imaging display device can be thereforereduced. In addition, by an AI unit disposed in the processing unit 102and using deep learning, accuracy and a processing speed in generatingone piece of image information from a plurality of pieces of imageinformation can be increased. In the present exemplary embodiment, afteran imaging unit is selected as illustrated in FIG. 12, image informationmay be acquired.

According to each of the above-described exemplary embodiments, it ispossible to obtain an imaging display device that reduces a differencebetween a displayed image and a real event.

While the disclosure has been described with reference to exemplaryembodiments, it is to be understood that the disclosure is not limitedto the disclosed exemplary embodiments. The scope of the followingclaims is to be accorded the broadest interpretation so as to encompassall such modifications and equivalent structures and functions.

In one embodiment, the processing units 102 and 1002, and the processingdevices 105 and 106 may be hardware circuits with specialized circuitelements to perform the operations described above (or below if thisparagraph is placed at the beginning). These circuits may beprogrammable logic devices (PLDs) or field programmable gate arrays(FPGAs), or applications-specific integrated circuits (ASICs), orsimilar devices. Alternatively, they may be programmable processors ordevices such as a central processing unit (CPU) to execute a program,instructions stored in memory devices to perform operations describedabove (or below if this paragraph is placed at the beginning).

This application claims the benefit of Japanese Patent Applications No.2019-121949, filed Jun. 28, 2019, and No. 2020-074083, filed Apr. 17,2020, which are hereby incorporated by reference herein in theirentirety.

What is claimed is:
 1. An imaging display device comprising: an imagingunit, including a plurality of photoelectric conversion elements,configured to acquire first image information; a processing unitconfigured to process the first image information from the imaging unitand generate second image information; a display unit configured todisplay an image that is based on the second image information from theprocessing unit; and a pupil detection unit configured to acquire vectorinformation on a pupil, wherein the processing unit generates the secondimage information by processing the first image information based on thevector information on the pupil.
 2. The imaging display device accordingto claim 1, wherein vector information on the imaging unit and the pupilincludes a distance between the imaging unit and the pupil.
 3. Theimaging display device according to claim 1, wherein the pupil detectionunit includes a function of detecting a line of sight, and wherein, whenprocessing the first image information, the processing unit sets aregion of interest in the first image information in accordance with thedetected line of sight, and performs processing of increasing resolutionof the region of interest.
 4. The imaging display device according toclaim 1, further comprising: a second imaging unit and a third imagingunit that are different from the imaging unit, wherein the pupildetection unit includes a function of detecting a line of sight, andwherein, based on the detected line of sight, the processing unit usesfirst image information on any of the imaging unit, the second imagingunit, and the third imaging unit.
 5. The imaging display deviceaccording to claim 1, wherein the imaging unit includes a backsideillumination complementary metal-oxide semiconductor (CMOS) imagesensor.
 6. The imaging display device according to claim 1, wherein theimaging unit includes a photon counting sensor.
 7. The imaging displaydevice according to claim 1, wherein the processing unit includes anartificial intelligence (AI) unit.
 8. The imaging display deviceaccording to claim 7, wherein the AI unit includes a deep learningfunction.
 9. The imaging display device according to claim 1, whereinthe imaging unit acquires the first image information at a first time,wherein, based on the first image information, the processing unitgenerates, as the second image information, first prediction imageinformation at a second time later than the first time, and wherein thedisplay unit displays an image that is based on the first predictionimage information, at the second time.
 10. The imaging display deviceaccording to claim 9, wherein the imaging unit acquires third imageinformation at the second time, and wherein, at a third time later thanthe second time, the imaging unit acquires fourth image information andthe display unit displays an image that is based on second predictionimage information generated from the third image information.
 11. Theimaging display device according to claim 10, wherein, at a fourth timebetween the first time and the second time, the imaging unit performs animage capturing operation of obtaining fifth image information, andwherein the first prediction image information is generated at leastfrom the first image information and the fifth image information. 12.The imaging display device according to claim 1, wherein the pluralityof photoelectric conversion elements can detect light in a visible lightregion and a near-infrared light region, and wherein the processing unitperforms processing of converting information in the near-infrared lightregion that is included in the first image information, into informationin the visible light region.
 13. The imaging display device according toclaim 1, wherein the display unit includes an organic light emittingdiode (LED) or an inorganic LED as a light emitting element.
 14. Theimaging display device according to claim 1, wherein, in the imagingunit, a substrate on which the plurality of photoelectric conversionelements is disposed, and a substrate on which a circuit configured toprocess a signal from the plurality of photoelectric conversion elementsis disposed are stacked.
 15. The imaging display device according toclaim 1, wherein at least three chips including a first chip on whichthe imaging unit is disposed, a second chip on which the display unit isdisposed, and a third chip on which the pupil detection unit is disposedare stacked.
 16. A wearable device comprising: the imaging displaydevice according to claim 1; and a power source unit configured tosupply power to the imaging display device.
 17. The wearable deviceaccording to claim 16, further comprising an interface unit configuredto perform wireless connection with an outside.
 18. An imaging displaydevice comprising: a first imaging unit, including a plurality ofphotoelectric conversion elements, configured to acquire first imageinformation; a second imaging unit, including a plurality ofphotoelectric conversion elements, configured to acquire second imageinformation; a processing unit configured to process the first imageinformation and the second image information, and generate third imageinformation; a display unit configured to display an image that is basedon the third image information from the processing unit; and a pupildetection unit configured to acquire vector information on a pupil,wherein the processing unit generates the third image information byprocessing the first image information and the second image informationbased on the vector information on the pupil.
 19. A wearable devicecomprising: the imaging display device according to claim 18; and apower source unit configured to supply power to the imaging displaydevice.
 20. The wearable device according to claim 19, furthercomprising an interface unit configured to perform wireless connectionwith an outside.